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ABSTRACT 

As the technology-enriched learning environments and theoretical constructs involved in instructional design 
become more sophisticated and complex, a need arises for equally sophisticated analytic methods to research 
these environments, theories, and models. Thus, this paper illustrates a comprehensive approach for analyzing 
data arising from experimental studies using structural equation modeling (SEM) procedures that can formulate 
and test theories regarding how interventions affect observed outcomes, in comparison to traditional MANCOVA 
design. Researchers in the field of instructional systems and educational technology are encouraged to 
incorporate this method into their analyses of experimental investigations, because this method allows for a close 
examination of mediating processes that are responsible for the outcomes observed and for the estimation of both 
random and correlated measurement errors. 

Keywords: Structural Equation Modeling, Experimental Research, Generative Learning, Latent Variable 

INTRODUCTION 

Conventionally, studies related to educational technology have used statistical techniques to test mean 
differences between groups. The t-test, analysis of variance (ANOVA), and analysis of covariance (ANCOVA) 
allow researcher to determine the effects of interventions. However, cognitive functioning and processes related 
to learning are intricate and human learning involves various psychological constructs. In other words, 
theoretical advances for understanding human cognition and learning processes require the consideration of more 
psychological constructs when designing learning environments. 

Moreover, with technological advances, educators infuse more technologies into learning environments to 
improve students’ learning. However, the effectiveness of these innovations cannot simply explained by testing 
mean differences, because these interventions could be related to underlying mediating processes that might be 
responsible for the desired outcomes (Koetting & Malisa, 2004, Delialioglu, et al., 2010). Thus, researchers in 
the field of educational technology should be more interested in explaining how interventions affect learning 
(Alenezi, Abdul Karim, &Veloo, 2010, Yukelturk, 2010). 

The traditional approach has been successful in finding the effectiveness of interventions, but not in 
understanding the intervening psychological constructs that might influence how an intervention affects learners’ 
achievement. Accordingly, the need arises for a more comprehensive approach that can formulate and test those 
complex mechanisms. In this paper, structural equation modeling is presented as a possible method. Although 
structural equation modeling has been used extensively in recent studies, most of the studies have used the 
method in non-experimental survey contexts. One reason could be that the procedures are relatively new and not 
easy to deploy in comparison to traditional methods such as ANOVA and MANOVA. For the same reason, there 
is no clear rationale for preferring structural equation modeling to traditional analyses of experimental data. 

Thus, the purpose of this paper is to demonstrate a comprehensive approach to analyzing data from experimental 
studies using latent variable structural equation modeling that can formulate and test theories regarding how 
interventions affect observed outcomes (Bollen, 1989; Kline, 2005). To illustrate this statistical approach, this 
paper analyzes data drawn from an actual experimental study. 

The Experimental Context 

To demonstrate the use of structural equation modeling for MANCOVA designs, this paper applies the 
procedures to data derived from an actual experiment that examined the effects of generative learning strategy 
prompts and metacognitive feedback on learners’ self-regulation, use of learning strategies, and learning 
performance (Lee, Lim, & Grabowski, 2010). In that experiment, the researchers wanted to create experimental 
conditions where interventions would influence students’ learning performance directly and indirectly. The 
prediction was that the interventions would positively affect learning performance, but through their effect on 
self-regulation and use of learning strategies. 

In the study, 223 participants were randomly assigned to one of the three treatment groups. One group was given 
only generative learning strategy tools as the control (Tl); the second group was given additional generative 
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learning strategy prompts (T2); and the third group was given additional generative learning strategy prompts 
and metacognitive feedbacks (T3). The participants took an online pre-test and were instructed to download the 
instructional material and study it. Afterwards, the participants completed a survey about their self-regulation 
while they were studying and took post-tests to explore two criteria: their recall and comprehension of the 
instructional material. The instructional materials that learners used during the experiment were collected and 
assessed to measure the quality of the learner’s overt use of generative learning strategies. 

In this experimental condition, researchers want to know whether the three groups differed significantly with 
respect to four dependent variables - learner’s self-regulation, quality of overt use of generative learning 
strategies, recall, and comprehension - when controlling learners’ prior knowledge. This question could be 
answered with a MANCOVA analysis. However, as instructional designers, we might be more interested in the 
mechanism among the four dependent variables. That is, researchers could hypothesize that metacognitive 
feedback would improve learners’ self-regulation and use of generative learning strategies, which in turn 
improve learners’ recall and comprehension. This mediational hypothesis could be tested by using latent variable 
structural equation modeling (hereafter SEM), but cannot be tested using traditional MANCOVA analysis. 

MODELING 

In applying structural equation modeling, researchers usually follow five basic steps of SEM recommended by 
Kline (2005): (1) Model Specification; (2) Model Identification; (3) Data Preparation and Screening; (4) 
Estimation of the Model; and (5) Model Re-specification, if necessary. Since the primary purpose of this paper is 
to demonstrate analyzing data from experimental studies with SEM, the general data analysis procedure of SEM 
is beyond the scope of this paper. Thus, the following sections present modeling one-way MANCOVA with a 
latent variable structural model, and alternative modeling to test mediating effects. 

Modeling One-Way MANCOVA with a Latent Variable Structural Model 

In general, researchers use MANOVA to test the mean differences across two or more groups on two or more 
dependent variables simultaneously. By using MANOVA, researchers control the overall alpha level, test mean 
differences while controlling independencies of dependent variables, and count the relationships among the 
dependent variables (Bray & Maxwell, 1985). These MANOVA designs can be accomplished by structural 
equation modeling using a Multiple Indicators and Multiple Causes (MIMIC) model. This SEM approach, in 
which factors are regressed on one or more dichotomous cause indicators that represent group membership (i.e. 
coded 0 = control, and 1 = treatment), allowed testing for multiple group differences on latent variables (Kaplan, 
2000), which is analogous to interconnected dummy variable regressions. 

As the one-way MANCOVA design shows, effects of treatments - generative learning strategy prompts and 
generative learning strategy prompts with metacognitive feedback - were tested after controlling for learner’s 
prior knowledge. Figure 1 presents the model used to examine the case with four latent dependent variables, 
three groups, and one covariate, adopting Kiihnel’s (1988) one-way MANOVA design and applying the LISREL 
notation (Joreskog & Shorbom, 1984). 
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The two causal indicators in the structural model of Figure 1 are two dichotomies using the group code (dummy 
code ) approach (Aiken, Stein, & Bentler, 1994). One dummy variable, Dummy 1 (gl), was coded as 1 = 
generative learning strategy prompts group (T2), or 0 = control group (Tl) or the generative learning strategy 
prompts with metacognitive feedback group (T3). This dummy variable, gl, represents the comparison between 
the generative learning strategy prompts group (T2) and the control group (Tl). Another dummy variable. 
Dummy 2 (g2), was coded as 1 = generative learning strategy prompts with metacognitive feedback group (T3), 
or 0 = control group (Tl) or the generative learning strategy prompts group (T2). This dummy variable, g2, 
represents the comparison between the generative learning strategy prompts with metacognitive feedback group 
(T3) and the control group (Tl). Table 1 shows this dummy code system. 


Table 1. Dummy codes for the MIMIC model 


Group 

Dummy 1 (gl) 

Dummy 2 (g2) 

Control group (Tl) 

0 

0 

Generative learning strategy prompts group (T2) 

1 

0 

Generative learning strategy prompts with metacognitive feedback 
group (T3) 

0 

1 


In order to analyze the means of observed dependent variables as a function of the categorical independent 
variables, a pseudo-variable (i.e. “CONST”) was added to the sample moment matrix as another variable, having 
1 in the diagonal and the means of all other variables as off-diagonal elements constant (“CONST”). The initial 
structural model analyzed the means of the observed dependent variables as a function of the categorical 
independent variables (Bagozzi & Yi, 1989). By doing this (see Figure 2), the first set of regression coefficients, 
Y *i, y ‘ 2 , y* 3 and j/* 4 , were the differences in the means of four dependent latent variables between the generative 
learning strategy prompts group and the control group. In the same way, the second set of regression coefficients, 
y ‘ ; ^* 3 and y* 4 , were the difference in the means of the four dependent latent variables between the 

generative learning strategy prompts with metacognitive feedback group and the control group. 
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Thus, an examination of the paths from the dummy exogenous variables to the dependent latent variables 
enabled testing of the multivariate null hypothesis: equality in means of the dependent variables across groups. 
This is analogous to the omnibus test commonly used in traditional MANCOVA analyses (e.g., the Pillai’s A or 
Wilks’ A). That is, if all regression coefficients from the dummy variables equal 0, then the null hypothesis - the 
means of dependent latent variables are equal across groups - is retained. In order to test the null hypothesis of 
equal means across groups, a chi-square difference test between a full model and the other restricted model 

(i.e.,/ n =,y l2 = ’Xu —and / |4 = 0and/,j =,Yn =,and = 0 ) can be conducted (Kaplan, 2000). 

As an illustration, the chi-square statistics of the full model, allowing for the difference in means as specified in 
Figure 2, and the restricted model, constraining the mean difference parameters to zero, appear in Table 2. The 
Satorra-Bentler scaled chi-square difference test (chi-square (8) = 164.00; p < .001) suggests rejecting the null 
hypothesis of equal means, as predicted. That is, generative learning strategy prompts and metacognitive 
feedback significantly affected learners’ self-regulation, use of learning strategies, and learning performance in 
one or more instances. 


Table 2. Chi-square statistics of the structural model for one-way MANCOVA 



X~NT 

p-value 

%~SB 

p-value 

df 

Full model 

78.10 

0.00 

78.26 

0.00 

30 

Restricted model 

220.77 

0.00 

227.36 

0.00 

38 


Note. % 2 NT : Normal theory weighted least squares chi-square 
X 2 sb: Satorra-Bentler scaled chi-square 
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After rejecting the null hypothesis, a significant test of each regression coefficient linking the dummy variables 
to the dependent latent variables allows researchers to examine which group affected which criteria. This test is 
analogous to the univariate ANOVA analysis of the dependent variables, but holds other variables in the model 
constant. To examine the univariate effect of the treatments, the regression coefficients from the dummy 
variables to the latent variables were inspected. Table 3 presents the unstandardized regression coefficient, 
standard error, and t-value. 


Table 3. Path coefficients, standard error, and t-value of treatments 


Latent variable 

Dummy 1 (gl): 

Control vs. generative learning 
strategy (GLS) prompts group 

Dummy 2 (g2): 

Control vs. GLS prompts with 
metacognitive feedback group 

Path coefficient 

t-value 

Path coefficient 

t-value 

Self-regulation 

.12 (.14) 

.87 

.25 (.019) 

1.97* 

The quality of overt use of GLS 

20.69(1.40) 

14.83* 

23.15 (1.35) 

17.09* 

Recall 

0.48 (.37) 

1.31 

1.14 (.37) 

3.03* 

Comprehension 

0.66 (.38) 

1.76 

1.42 (.40) 

3.52* 


Note. Standard errors are in parenthesis 
*: p < .05 


According to the previous results, five significant paths were identified, linking treatments to four dependent 
variables, including from Dummy 1 (gl) to USE and from Dummy 2 (g2) to four dependent variables (see 
Figure 2). Conversely, three paths, linking Dummy 1 to self-regulation, recall, and comprehension, were not 
significant. Thus, these insignificant paths were removed and the modified structural model was estimated with 
only statistically significant paths, as recommended by Kline (2005) and Kaplan (2000) (see Figure 3). The 
modified model obtained a significant chi-square (x2SB=82.17; df =33; p < .000), the CFI = .95, the RMSEA 
= .082, and the SRMR = .016. Although the chi-square was significant and RMSEA was slightly greater than the 
criteria (.06), other fit indices satisfied the criteria, suggesting acceptable model fit. 



Chi“Square=82.17, d£=33, P-value=0.00000, FMSEA=0.082 


Note, gl: dummy code for T2 group, g2: dummy code for T3 group, pre: prior knowledge, 

SR: self-regulation, cog: cognitive control, meta: metacognitive control, 

USE: the quality of overt use of generative learning strategies, highl: highlighting, note: note taking, 
REC: recall, fact: fact, termi: terminology, 

COM: comprehension, cone: concept, relat: relationship 
_ Figure 3. Structural model with significant paths _ 
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Alternative Model to Test Mediating Effects 

After researchers confirm that groups differ on variables, they may want to know if those differences are directly 
affected by the interventions, or indirectly as a result of a causal ordering among variables. For example, the 
second research question of the study examined whether or not variation in learners’ performance in recall and 
comprehension was due to a direct association with the treatments, its dependence on learners’ self-regulation, or 
overt use of generative learning strategies. Thus, three paths, linking self-regulation to learners’ use of generative 
learning strategies and learners’ use of generative learning strategies to recall and comprehension, replaced the 
error covariance among them, as hypothesized (see Figure 4). 



CKi-3<iuare = 100.72, d£=33, P-value=0.00000, BH3EA=0.096 


Figure 4. Structural equation model with hypothesized causal paths 


To test the mediational hypothesis, the scaled chi-square difference between the model (see Figure 3), which 
includes the direct effects of generative learning strategy prompts with metacognitive feedback (g2) on recall and 
comprehension and the model which does not include these direct paths (i.e., y* = y‘ t = 0), was tested. The 

chi-square statistics of these two models appear in Table 4. 


Table 4. Chi-square statistics of the structural model with causal paths 



X 2 nt 

p-value 

X~SB 

p-value 

df 

Full model (Model 5) 

101.72 

0.00 

100.72 

0.00 

33 

Model with / 23 = y 04 - 0 

102.69 

0.00 

101.62 

0.00 

35 


Note. jfNF Normal theory weighted least squares chi-square 
% 2 SB : Satorra-Bentler scaled chi-square 


The Satorra-Bentler scaled chi-square difference test (chi-square (2) = .95, p > .05) suggests that retaining the 
null hypothesis of the direct effects of generative learning strategy prompts metacognitive feedback on recall and 
comprehension, thus supporting the indirect effects of self-regulation and learners’ use of generative learning 
strategies. That is, the improvement of learners, who received generative learning strategy prompts with 
metacogntive feedback over the control group learners on recall and comprehension, can be explained with the 
improvement of the quality of their overt use of generative learning strategies. Also, self-regulation had 
significant, indirect effects on recall and comprehension through the quality of overt use of generative learning 
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strategies, supporting the mediation effect of learners’ self-regulation and their use of generative learning 
strategies. The final structural equation model of the study is shown in Figure 5. 



Chi-S<iuar«=65.56, d£=31, P-valvie=0.00028, FMSEA=0.0tl 

Note, gl: dummy code for T2 group, g2: dummy code for T3 group, pre: prior knowledge, 

SR: self-regulation, cog: cognitive control, meta: metacognitive control. 

USE: the quality of overt use of generative learning strategies, highl: highlighting, note: note 
taking, 

REC: recall, fact: fact, termi: terminology, 

COM: comprehension, cone: concept, relat: relationship 
Note, the disturbances between SR and REC, SR and COM, and REC and COM are correlated 


Figure 5. Final structural equation model 


DISCUSSION 

The major goal of instructional systems or educational technology is to design learning environments, providing 
meaningful instructional interventions to help learners. Accordingly, examining the effectiveness of the 
instructional interventions is a prime concern of the research in this field (Koetting & Malisa, 2004). 
Conventionally, studies have used statistical techniques to test mean differences between groups, such as the t- 
test, analysis of variance (ANOVA), and analysis of covariance (ANCOVA), to determine the effects of 
interventions. However, as current technology-enriched learning environments and theoretical constructs 
involved in instructional design and development become more sophisticated and more complex, a need arises 
for equally sophisticated analytic methods to research these environments, theories, and models. This paper 
demonstrated a comprehensive statistical analysis, structural equation modeling (SEM), which is a methodology 
that combines factor analysis and path analysis (Bollen, 1989; Kline, 2005; Russell, Kahn, & Altmaier, 1998). 
This SEM approach can be used to answer the research questions, exploring how interventions affect learning 
and examining the indirect effect of related psychological constructs. 

Although educational studies have used this SEM approach extensively, the majority has used this method for 
analyzing non-experimental survey data. The advantages of using SEM with experimental data over traditional 
MANOVA/MANCOVA analyses are: 1) estimating and removing both random and correlated measurement 
errors; and 2) examining mediating processes (Bagozzi & Yi, 1989; Kahn & Altmaier, 1998). First, the 
traditional MANOVA/MANCOVA analysis assumes that dependent variables have no measurement errors. 
Ignoring the measurement errors of dependent variables increases the chances of making Type II errors, whereas 
SEM uses latent variables, which allows estimation and corrects the measurement errors. As a result, the latent 
variable SEM approach estimates the experimental intervention effects more accurately than traditional methods 
(Kahn & Altmaier, 1998). Second, as this study formulated, SEM can test factors that hypothesize the mediation 
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treatment effects on the dependent variables. This allows the uncovering of underlying processes of treatment 
influences. Obviously, the traditional approach has been successful in finding the effectiveness of interventions, 
but not in understanding how the interventions are effective. Analyses of the processes underlying a treatment 
might allow researchers to design more effective instructional treatments by refining the treatments to focus on 
processes that are positively related to treatment outcome (Kahn & Altmaier, 1998). 

However, two important issues need to be addressed before applying this alternative procedure. First, a path 
analysis in SEM involves the estimation of causal relations among variables with correlational data. However, 
correlation does not imply causation, thereby not enabling statistical causal modeling to prove causation either. 
Inferring causation from correlation requires a solid theoretical base and careful specification of variables and 
predictive directions. For example, this study hypothesized that learners’ self-regulation would cause improved 
overt use of generative learning strategies with theoretical basis; the final model (see Figure 4) supported this 
hypothesis. However, an alternative hypothesis that predicts effects from the opposite direction (i.e. from 
learners’ overt use of generative learning strategies to learners’ self-regulation) is a possible equivalent model. 
This alternative model obtained worse model fit indices (x 2 sb=72.06; df =31', p=.000; CFI = .97; RMSEA =.077; 
and SRMR =.043) than the final model. Thus, the direction from learners’ self-regulation to learners' overt use 
of generative learning strategies was supported by the model, but caution is still advised; SEM itself does not 
prove any causal relationships. 

Second, an SEM analysis with latent variables needs more than 200 cases to produce accurate estimates (Kline, 
2005). Also, researchers should consider the number of parameters being estimated as well as sample size. 
Bender and Chou (1988) suggested that the ratio of participants to parameters should be at least 5:1 for 
appropriate estimation. Obtaining the appropriate number of research participants can often be challenging for 
educational researchers. 

Cognitive functioning and processes related to learning are intricate and human learning involves various 
psychological constructs. Theoretical advances for understanding human cognition and learning processes 
require the consideration of more psychological constructs when designing learning environments. In addition, 
technological advances allows educators to infuse more technologies into learning environments, but testing 
mean differences cannot explain the effectiveness of these innovations; underlying mediating processes are more 
responsible for the desired outcomes. 

Therefore, researchers in the field of instructional systems and educational technology are encouraged to assess 
learners’ interaction with instructional interventions in technology-enriched environments. Current technologies 
permit developing user-oriented Web instructions that allow users to manipulate Web pages and record all 
learners’ activities during their interactions, such as times of visits and revisions of notes. In this way, future 
investigations might reveal how learners interact with instructional interventions and how these interactions 
affect their learning. Thus, incorporating structural equation modeling for the analyses of experimental 
investigation is recommended. This method informs instructional designers about the direct and indirect effects 
of instructional interventions, and how intervening psychological constructs affect learning, rather than focusing 
only on the direct effects. Accordingly, this method enables instructional designers to identify problems in the 
treatment mechanisms and implement appropriate treatment. 

CONCLUSION 

The latent variable structural equation modeling approach has a major advantage over traditional analyses in that 
the comprehensive approach allows for a close examination of the mediating processes responsible for the 
observed outcomes. The traditional approach has been successful in finding the effectiveness of interventions, 
but not in understanding why the interventions were successful. Analyses of the processes underlying an 
intervention may allow researchers to design more effective instructional treatments by refining the intervention 
to focus on processes that are positively related to treatment outcome. 

In addition, the SEM approach using latent variable modeling procedures allows for the estimation of both 
random and correlated measurement errors. As a result, the SEM approach provides more accurate estimates of 
the effects of experimental interventions than traditional approaches that ignore measurement errors of 
dependent variables that increase the chance of making Type II errors. Even though this alternative approach 
requires a larger sample size than the traditional approach, researchers in the field of educational technology are 
encouraged to incorporate this method into their analyses of experimental investigation, since this method allows 
them to analyze more sophisticated and advanced future learning environments. 
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